Input parameter based image waves

ABSTRACT

A virtual wave creation system comprises an eyewear device that includes a frame, a temple connected to a lateral side of the frame, and a depth-capturing camera. Execution of programming by a processor configures the virtual wave creation system to generate, for each of multiple initial depth images, a respective wave image by applying a transformation function that is responsive to a selected input parameter to the initial three-dimensional coordinates. The virtual wave creation system creates a warped wave video including a sequence of the generated warped wave images. The virtual wave creation system presents, via an image display, the warped wave video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. Application Serial No.17/526,136 filed on Nov. 15, 2021, which is a Continuation of U.SApplication Serial No. 16/658,370 filed on Oct. 21, 2019, now U.S. Pat.No. 11,178,375, the contents of both of which are incorporated fullyherein by reference.

TECHNICAL FIELD

The present subject matter relates to wearable devices, e.g., eyeweardevices, and mobile devices and techniques to allow a user to createwaves based on an input parameter (such as audio signal or a heartbeat)in three-dimensional space in videos and images.

BACKGROUND

Computing devices, such as wearable devices, including portable eyeweardevices (e.g., smartglasses, headwear, and headgear); mobile devices(e.g., tablets, smartphones, and laptops); and personal computersavailable today integrate image displays and cameras. Currently, usersof computing devices can utilize photo filters to create effects ontwo-dimensional (2D) photographs. Various photo decorating applicationsfeature tools like stickers, emojis, and captions to edittwo-dimensional photographs.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations, by way ofexample only, not by way of limitations. In the figures, like referencenumerals refer to the same or similar elements.

FIG. 1A is a right side view of an example hardware configuration of aneyewear device utilized in a virtual wave creation system, in which atransformation function is applied to initial depth images of an initialvideo to generate warped wave images to create a warped wave video.

FIG. 1B is a top cross-sectional view of a right chunk of the eyeweardevice of FIG. 1A depicting a right visible light camera of adepth-capturing camera, and a circuit board.

FIG. 1C is a left side view of an example hardware configuration of aneyewear device of FIG. 1A, which shows a left visible light camera ofthe depth-capturing camera.

FIG. 1D is a top cross-sectional view of a left chunk of the eyeweardevice of FIG. 1C depicting the left visible light camera of thedepth-capturing camera, and the circuit board.

FIG. 2A is a right side view of another example hardware configurationof an eyewear device utilized in the virtual wave creation system, whichshows the right visible light camera and a depth sensor of thedepth-capturing camera to generate an initial depth image of a sequenceof initial depth images (e.g., in an initial video).

FIGS. 2B and 2C are rear views of example hardware configurations of theeyewear device, including two different types of image displays.

FIG. 3 shows a rear perspective sectional view of the eyewear device ofFIG. 2A depicting an infrared camera of the depth sensor, a frame front,a frame back, and a circuit board.

FIG. 4 is a cross-sectional view taken through the infrared camera andthe frame of the eyewear device of FIG. 3 .

FIG. 5 shows a rear perspective view of the eyewear device of FIG. 2Adepicting an infrared emitter of the depth sensor, the infrared cameraof the depth sensor, the frame front, the frame back, and the circuitboard.

FIG. 6 is a cross-sectional view taken through the infrared emitter andthe frame of the eyewear device of FIG. 5 .

FIG. 7 depicts an example of a pattern of infrared light emitted by theinfrared emitter of the depth sensor and reflection variations of theemitted pattern of infrared light captured by the infrared camera of thedepth sensor of the eyewear device to measure depth of pixels in a rawimage to generate the initial depth images from the initial video.

FIG. 8A depicts an example of infrared light captured by the infraredcamera of the depth sensor as an infrared image and visible lightcaptured by a visible light camera as a raw image to generate theinitial depth image of a three-dimensional scene.

FIG. 8B depicts an example of visible light captured by the left visiblelight camera as left raw image and visible light captured by the rightvisible light camera as a right raw image to generate the initial depthimage of a three-dimensional scene.

FIG. 9 is a high-level functional block diagram of an example virtualwave creation system including the eyewear device with a depth-capturingcamera to generate the initial depth images (e.g., in the initial video)and a user input device (e.g., touch sensor), a mobile device, and aserver system connected via various networks.

FIG. 10 shows an example of a hardware configuration for the mobiledevice of the virtual wave creation system of FIG. 9 , which includes auser input device (e.g., touch screen device) to receive the wave effectselection to apply to the initial depth images to generate warped waveimages (e.g., in the warped wave video).

FIG. 11 is a flowchart of a method that can be implemented in thevirtual wave creation system to apply waves to the initial depth imagesfrom the initial video to generate the warped wave images to create thewarped wave video.

FIGS. 12A and 12B illustrate an example of a raw image captured by oneof the visible light cameras and application of a transformationfunction to a wave region of vertices of a generated initial depthimage, respectively.

FIGS. 13A and 13B illustrate an example of a raw image captured by oneof the visible light cameras and application of a transformationfunction that is responsive to an input parameter (audio track) to aregion of vertices in a generated initial depth image, respectively.

FIG. 14 illustrates another example of a transformation function that isresponsive to an input parameter (audio track) to a region of verticesin a generated initial depth image; and

FIG. 15 illustrates incorporation of the virtual wave creation system inaugmented reality eyewear.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, description of well-known methods,procedures, components, and circuitry are set forth at a relativelyhigh-level, without detail, in order to avoid unnecessarily obscuringaspects of the present teachings.

As used herein, the term “wave” means a computer-generated effectapplied to an image or series of images that creates the appearance ofwaves traveling through a medium(s), such as a structure, person, and/orair. The term “coupled” or “connected” as used herein refers to anylogical, optical, physical or electrical connection, link or the like bywhich electrical or magnetic signals produced or supplied by one systemelement are imparted to another coupled or connected element. Unlessdescribed otherwise, coupled or connected elements or devices are notnecessarily directly connected to one another and may be separated byintermediate components, elements or communication media that maymodify, manipulate or carry the electrical signals. The term “on” meansdirectly supported by an element or indirectly supported by the elementthrough another element integrated into or supported by the element.

The orientations of the eyewear device, associated components and anycomplete devices incorporating a depth-capturing camera such as shown inany of the drawings, are given by way of example only, for illustrationand discussion purposes. In operation for wave creation, the eyeweardevice may be oriented in any other direction suitable to the particularapplication of the eyewear device, for example up, down, sideways, orany other orientation. Also, to the extent used herein, any directionalterm, such as front, rear, inwards, outwards, towards, left, right,lateral, longitudinal, up, down, upper, lower, top, bottom, side,horizontal, vertical, and diagonal are used by way of example only, andare not limiting as to direction or orientation of any depth-capturingcamera or component of the depth-capturing camera constructed asotherwise described herein.

Additional objects, advantages and novel features of the examples willbe set forth in part in the following description, and in part willbecome apparent to those skilled in the art upon examination of thefollowing and the accompanying drawings or may be learned by productionor operation of the examples. The objects and advantages of the presentsubject matter may be realized and attained by means of themethodologies, instrumentalities and combinations particularly pointedout in the appended claims.

Reference now is made in detail to the examples illustrated in theaccompanying drawings and discussed below.

FIG. 1A is a right side view of an example hardware configuration of aneyewear device 100 utilized in a virtual wave creation system, whichshows a right visible light camera 114B of a depth-capturing camera togenerate an initial depth image. As further described below, in thevirtual wave creation system, a transformation function is applied to asequence of initial depth images of an initial video to generate thesequence of warped wave images of a warped wave video. Thistransformation function is responsive to an input parameter (e.g., anaudio input or a biometric input such as heart rate) and can depend onspatial and temporal coordinates of the initial depth images, asexplained below.

Eyewear device 100, includes a right optical assembly 180B with an imagedisplay to present an initial video including initial images and warpedwave images of a warped wave video that are two-dimensional. In theexample, the initial video includes initial images or processed rawimages that are presented to the user, but a depth video that includesthe initial depth images generated based on processed raw images is notpresented to the user. Instead, a warped version of the image ispresented to the user. In the example, this depth video that includesthe generated initial depth images is used for purposes of calculationsto generate the warped wave images and create the warped wave video.

As shown in FIGS. 1A and 1B, the eyewear device 100 includes the rightvisible light camera 114B. Eyewear device 100 can include multiplevisible light cameras 114A and 114B that form a passive type ofdepth-capturing camera, such as stereo camera, of which the rightvisible light camera 114B is located on a right chunk 110B. As shown inFIGS. 1C and 1D, the eyewear device 100 can also include a left visiblelight camera 114A. Alternatively, in the example of FIG. 2A, thedepth-capturing camera can be an active type of depth-capturing camerathat includes a single visible light camera 114B and a depth sensor (seeelement 213 of FIG. 2A).

Left and right visible light cameras 114A and 114B are sensitive to thevisible light range wavelength. Each of the visible light cameras 114Aand 114B have a different frontward facing field of view which areoverlapping to allow three-dimensional depth images to be generated, forexample, right visible light camera 114B has the depicted right field ofview 111B. Generally, a “field of view” is the part of the scene that isvisible through the camera at a particular position and orientation inspace. Objects or object features outside the field of view 111A and111B when the image is captured by the visible light camera are notrecorded in a raw image (e.g., photograph or picture). The field of viewdescribes an angle range or extent which the image sensor of the visiblelight camera 114A and 114B picks up electromagnetic radiation of a givenscene in a captured image of the given scene. Field of view can beexpressed as the angular size of the view cone, i.e., an angle of view.The angle of view can be measured horizontally, vertically, ordiagonally.

In an example, visible light cameras 114A and 114B have a field of viewwith an angle of view between 15° to 30°, for example 24°, and have aresolution of 480 × 480 pixels. The “angle of coverage” describes theangle range that a lens of visible light cameras 114A and 114B orinfrared camera 220 (see FIG. 2A) can effectively image. Typically, theimage circle produced by a camera lens is large enough to cover the filmor sensor completely, possibly including some vignetting (i.e., areduction of an image’s brightness or saturation at the peripherycompared to the image center). If the angle of coverage of the cameralens does not fill the sensor, the image circle will be visible,typically with strong vignetting toward the edge, and the effectiveangle of view will be limited to the angle of coverage.

Examples of such visible lights camera 114A and 114B include ahigh-resolution complementary metal-oxide-semiconductor (CMOS) imagesensor and a video graphic array (VGA) camera, such as 640p (e.g., 640 ×480 pixels for a total of 0.3 m 3egapixels), 720p, or 1080p. As usedherein, the term “overlapping” when referring to field of view means thematrix of pixels in the generated raw image(s) or infrared image of ascene overlap by 30% or more. As used herein, the term “substantiallyoverlapping” when referring to field of view means the matrix of pixelsin the generated raw image(s) or infrared image of a scene overlap by50% or more.

Image sensor data from the visible light cameras 114A and 114B arecaptured along with geolocation data, digitized by an image processor,and stored in a memory. The captured left and right raw images capturedby respective visible light cameras 114A and 114B are in thetwo-dimensional space domain and comprise a matrix of pixels on atwo-dimensional coordinate system that includes an X axis for horizontalposition and a Y axis for vertical position. Each pixel includes a colorattribute (e.g., a red pixel light value, a green pixel light value,and/or a blue pixel light value); and a position attribute (e.g., an Xlocation coordinate and a Y location coordinate).

To provide stereoscopic vision, visible light cameras 114A and 114B maybe coupled to an image processor (element 912 of FIG. 9 ) for digitalprocessing along with a timestamp in which the image of the scene iscaptured. Image processor 912 includes circuitry to receive signals fromthe visible light cameras 114A and 114B and process those signals fromthe visible light camera 114 into a format suitable for storage in thememory. The timestamp can be added by the image processor or otherprocessor, which controls operation of the visible light cameras 114Aand 114B. Visible light cameras 114A and 114B allow the depth-capturingcamera to simulate human binocular vision. Depth-capturing cameraprovides the ability to reproduce three-dimensional images based on twocaptured images from the visible light cameras 114A and 114B having thesame timestamp. Such three-dimensional images allow for an immersivelife-like experience, e.g., for virtual reality or video gaming.Three-dimensional depth videos may be produced by stitching together asequence of three-dimensional depth images with associated timecoordinates for a presentation time in a depth video.

For stereoscopic vision, a pair of raw red, green, and blue (RGB) imagesare captured of a scene at a moment in time - one image for each of theleft and right visible light cameras 114A and 114B. When the pair ofcaptured raw images from the frontward facing left and right field ofviews 111A and 111B of the left and right visible light cameras 114A and114B are processed (e.g., by the image processor), depth images aregenerated, and the generated depth images can be perceived by a user onthe optical assembly 180A and 180B or other image display(s) (e.g., of amobile device). The generated depth images are in the three-dimensionalspace domain and can comprise a matrix of vertices on athree-dimensional location coordinate system that includes an X axis forhorizontal position (e.g., length), a Y axis for vertical position(e.g., height), and a Z axis for a depth position (e.g., distance).

A depth video further associates each of a sequence of generated depthimages with a time coordinate on a time (T) axis for a presentation timein a depth video (e.g., each depth image includes spatial components aswell as a temporal component). The depth video can further include oneor more input parameter components (e.g., an audio component such as anaudio track or stream, a biometric comp such as a heartrate graph,etc.), which may be captured by an input device such as a microphone ora heartrate monitor. Each vertex includes a color attribute (e.g., a redpixel light value, a green pixel light value, and/or a blue pixel lightvalue); a position attribute (e.g., an X location coordinate, a Ylocation coordinate, and a Z location coordinate); a texture attribute,and/or a reflectance attribute. The texture attribute quantifies theperceived texture of the depth image, such as the spatial arrangement ofcolor or intensities in a region of vertices of the depth image.

Generally, perception of depth arises from the disparity of a given 3Dpoint in the left and right raw images captured by visible light cameras114A and 114B. Disparity is the difference in image location of the same3D point when projected under perspective of the visible light cameras114A and 114B (d = x_(left) - x_(right)). For visible light cameras 114Aand 114B with parallel optical axes, focal length f, baseline b, andcorresponding image points (x_(left), y_(left)) and (x_(right),y_(right)), the location of a 3D point (Z axis location coordinate) canbe derived utilizing triangulation which determines depth fromdisparity. Typically, depth of the 3D point is inversely proportional todisparity. A variety of other techniques can also be used. Generation ofthree-dimensional depth images and warped wave images is explained inmore detail later.

In an example, a virtual wave creation system includes the eyeweardevice 100. The eyewear device 100 includes a frame 105 and a lefttemple 110A extending from a left lateral side 170A of the frame 105 anda right temple 110B extending from a right lateral side 170B of theframe 105. Eyewear device 100 further includes a depth-capturing camera.The depth-capturing camera includes: (i) at least two visible lightcameras with overlapping fields of view; or (ii) a least one visiblelight camera 114A and 114B and a depth sensor (element 213 of FIG. 2A).In one example, the depth-capturing camera includes a left visible lightcamera 114A with a left field of view 111A connected to the frame 105 orthe left temple 110A to capture a left image of the scene. Eyeweardevice 100 further includes a right visible light camera 114B connectedto the frame 105 or the right temple 110B with a right field of view111B to capture (e.g., simultaneously with the left visible light camera114A) a right image of the scene which partially overlaps the leftimage.

Virtual wave creation system further includes a computing device, suchas a host computer (e.g., mobile device 990 of FIGS. 9 and 10 ) coupledto eyewear device 100 over a network. The virtual wave creation systemfurther includes an image display (optical assembly 180A and 180B ofeyewear device; image display 1080 of mobile device 990 of FIG. 10 ) forpresenting (e.g., displaying) a video including images. Virtual wavecreation system further includes an image display driver (element 942 ofeyewear device 100 of FIG. 9 ; element 1090 of mobile device 990 of FIG.10 ) coupled to the image display (optical assembly 180A and 180B ofeyewear device; image display 1080 of mobile device 990 of FIG. 10 ) tocontrol the image display to present the initial video. Virtual wavecreate system further includes an input parameter processing module(element 910 of eyewear device 100 of FIG. 9 ; elements 1092 of mobiledevice 990 of FIG. 10 ) to produce a signal (e.g., based on an audiotrack or a heart rate graph) for use in wave creation.

In some examples, user input is received to indicate that the userdesires waves to be applied to the various initial depth images from theinitial video. For example, virtual wave creation system furtherincludes a user input device to receive a wave effect selection from auser to apply waves to the presented initial video based on an inputparameter such as a desired audio or their heart rate. Examples of userinput devices include a touch sensor (element 991 of FIG. 9 for theeyewear device 100), a touch screen display (element 1091 of FIG. 10 forthe mobile device 990), and a computer mouse for a personal computer ora laptop computer. Virtual wave creation system further includes aprocessor (element 932 of eyewear device 100 of FIG. 9 ; element 1030 ofmobile device 990 of FIG. 10 ) coupled to the eyewear device 100 and thedepth-capturing camera. Virtual wave creation system further includes amemory (element 934 of eyewear device 100 of FIG. 9 ; elements 1040A-Bof mobile device 990 of FIG. 10 ) accessible to the processor, and wavecreation programming in the memory (element 945 of eyewear device 100 ofFIG. 9 ; element 945 of mobile device 990 of FIG. 10 ), for example inthe eyewear device 100 itself, mobile device (element 990 of FIG. 9 ),or another part of the virtual wave creation system (e.g., server system998 of FIG. 9 ).

Execution of the programming (element 945 of FIG. 9 ) by the processor(element 932 of FIG. 9 ) configures the eyewear device 100 to generate,via the depth-capturing camera, the initial depth images from theinitial images 957A-N in the initial video. The initial images 957A-Nare in two-dimensional space, for example raw images 858A-B or processedraw images 858A-B after rectification. Each of the initial depth imagesis associated with a time coordinate on a time (T) axis for apresentation time, for example based, on initial images 957A-B in theinitial video. The initial depth image is formed of a matrix ofvertices. Each vertex represents a pixel in a three-dimensional scene.Each vertex has a position attribute. The position attribute of eachvertex is based on a three-dimensional location coordinate system andincludes an X location coordinate on an X axis for horizontal position,a Y location coordinate on a Y axis for vertical position, and a Zlocation coordinate on a Z axis for a depth position.

Execution of the wave creation programming (element 945 of FIG. 10 ) bythe processor (element 1030 of FIG. 10 ) configures the mobile device(element 990 of FIG. 10 ) of the virtual wave creation system to performthe following functions. Mobile device (element 990 of FIG. 10 )presents, via the image display (element 1080 of FIG. 10 ), the initialvideo. Mobile device (element 990 of FIG. 10 ) receives, via the userinput device (element 1091 of FIG. 10 ), the wave effect selection fromthe user to apply waves to the presented initial video. In response toreceiving the wave effect selection, based on, at least, the associatedtime coordinate of each of the initial depth images and at least oneinput parameter such as an audio-based signal and/or a heart rate,mobile device (element 990 of FIG. 10 ) applies to vertices of each ofthe initial depth images, a transformation function.

The transformation function can transform a respective wave region ofvertices grouped together along the Z axis based on, at least, theassociated time coordinate of the respective initial depth image. Therespective transformation function moves a respective Y locationcoordinate of vertices in the respective wave region of verticesvertically upwards or downwards on the Y axis, which appears as a depthwarping effect. Additionally, the transformation function may include acolor component that alters the color of respective vertices responsiveto the input parameter. The transformation function may alter the Ylocation coordinate based on one parameter (e.g., audio track), thevertice color based on another parameter (e.g., heart rate), or anycombination thereof. In one example, based on the X, Y, and/or Zlocation coordinates of the vertices, the associated time coordinate,and the input parameter-based signal, the transformation functiontransforms all the vertices of the initial depth images. In one examplethe transformation functions is an equation new Y = func (X, old Y,Z,T),which does not depend on the X location coordinate. An exampletransformation function where the wave is advancing in the Z direction(so this specific function does not depend on X) is new Y = f (Y,Z,t) =Y + 200/(exp(20/3 - abs(Z - 300*t)/150) + 1) - 200. The transformationis applied per-vertex, where the transformation relies on both space andtime. Applying the transformation function creates a new modified set ofvertices or a three-dimensional image without a depth map.

The transformation function incorporates an input parameter-based signal(e.g., an audio track, heart rate graph, etc.) and an epicenterselection. An audio-based signal is generated from an audio track, whichmay be a prerecorded song or audio captured via a microphone. Abiometric based signal is generated from a measured biometric such asheart rate obtained from a heart rate monitor. In an audio-basedexample, the transformation function may produce each wave by callinginput processing module 910/1092 (e.g., an audio processing module) foraudio samples. Each callback from the input processing module returns Nsamples (e.g., 4096). The transformation function averages the samplesto produce an average amplitude that, together with a timestamp, marksone wave. This wave is added to a buffer of waves (e.g., stored inmemory, with the oldest wave deleted when the buffer is full). Thevector of waves is then parsed, and the contribution of each wave isadded to each vertex of the parsed waves in the vertex matrix. The sizeof the buffer of waves can be, for example, 128. An example for such acontribution is ΔY=f(distance from the epicenter, amplitude, time) =0.01*amplitude*(1.0-1.0/(1.0+exp(-(abs(distance from theepicenter*10.0-time*2.5)-100.0/55.0))). In another example, the wavesare created using the frequency domain by applying a Discrete FoyerTransform (DFT) to the wave samples, e.g., using graphical equalizertechniques. Similar techniques for processing other types of inputparameter signals such as biometric signals will be understood by one ofskill the art from the description herein. In an example, the epicenteris a predefined location, e.g., lower middle location in the images towhich the waves are being applied. In another example, the epicenter maybe selected and optionally moved by a user, e.g., by selecting alocation with an input device. The selected location may be associatedwith an image a cartoon character, Bitmoji®, etc.

Mobile device (element 990 of FIG. 10 ) generates, for each of theinitial depth images, a respective wave depth image by applying thetransformation function to the respective initial depth image. Mobiledevice (element 990 of FIG. 10 ) can generate a respective wave depthimage by applying the transformation function to the position attributeof the vertices in a respective wave region of vertices of therespective initial depth image. Mobile device (element 990 of FIG. 10 )creates, a warped wave video including the sequence of the generatedwarped wave images. Mobile device (element 990 of FIG. 10 ) presents,via the image display (image display 1080 of FIG. 10 ), the warped wavevideo. Mobile device (element 990 of FIG. 10 ) presents, via the imagedisplay (image display 1080 of FIG. 10 ), the warped wave video. Variouswave creation programming (element 945 of FIGS. 9-10 ) functionsdescribed herein may be implemented within other parts of the virtualwave creation system, such as the eyewear device 100 or another hostcomputer besides mobile device (element 990 of FIG. 10 ), such as aserver system (element 998 of FIG. 9 ).

In some examples, the received wave effect selection generates a wavecreation photo filter effect, which is applied as the transformationfunction to the initial video, an input parameter such as an audiotrack, and a finger moving on a touch screen display (e.g., combinedimage display 1080 and user input device 1091). The warped wave videowith the wave creation photo filter effect may then be shared withfriends via a chat application executing on the mobile device (element990 of FIG. 10 ) by transmission over a network.

FIG. 1B is a top cross-sectional view of a right chunk 110B of theeyewear device 100 of FIG. 1A depicting the right visible light camera114B of the depth-capturing camera, and a circuit board. FIG. 1C is aleft side view of an example hardware configuration of an eyewear device100 of FIG. 1A, which shows a left visible light camera 114A of thedepth-capturing camera. FIG. 1D is a top cross-sectional view of a leftchunk 110A of the eyewear device of FIG. 1C depicting the left visiblelight camera 114A of the depth-capturing camera, and a circuit board.Construction and placement of the left visible light camera 114A issubstantially similar to the right visible light camera 114B, except theconnections and coupling are on the left lateral side 170A. As shown inthe example of FIG. 1B, the eyewear device 100 includes the rightvisible light camera 114B and a circuit board, which may be a flexibleprinted circuit board (PCB) 140B. The right hinge 226B connects theright chunk 110B to a right temple 125B of the eyewear device 100. Insome examples, components of the right visible light camera 114B, theflexible PCB 140B, or other electrical connectors or contacts may belocated on the right temple 125B or the right hinge 226B.

The right chunk 110B includes chunk body 211 and a chunk cap, with thechunk cap omitted in the cross-section of FIG. 1B. Disposed inside theright chunk 110B are various interconnected circuit boards, such as PCBsor flexible PCBs, that include controller circuits for right visiblelight camera 114B, microphone(s), low-power wireless circuitry (e.g.,for wireless short range network communication via Bluetooth™),high-speed wireless circuitry (e.g., for wireless local area networkcommunication via WiFi).

The right visible light camera 114B is coupled to or disposed on theflexible PCB 140B and covered by a visible light camera cover lens,which is aimed through opening(s) formed in the frame 105. For example,the right rim 107B of the frame 105 is connected to the right chunk 110Band includes the opening(s) for the visible light camera cover lens. Theframe 105 includes a front-facing side configured to face outwards awayfrom the eye of the user. The opening for the visible light camera coverlens is formed on and through the front-facing side. In the example, theright visible light camera 114B has an outward facing field of view 111Bwith a line of sight or perspective of the right eye of the user of theeyewear device 100. The visible light camera cover lens can also beadhered to an outward facing surface of the right chunk 110B in which anopening is formed with an outward facing angle of coverage, but in adifferent outward direction. The coupling can also be indirect viaintervening components.

Left (first) visible light camera 114A is connected to a left imagedisplay of left optical assembly 180A to capture a left eye viewed sceneobserved by a wearer of the eyewear device 100 in a left raw image.Right (second) visible light camera 114B is connected to a right imagedisplay of right optical assembly 180B to capture a right eye viewedscene observed by the wearer of the eyewear device 100 in a right rawimage. The left raw image and the right raw image partially overlap topresent a three-dimensional observable space of a generated depth image.

Flexible PCB 140B is disposed inside the right chunk 110B and is coupledto one or more other components housed in the right chunk 110B. Althoughshown as being formed on the circuit boards of the right chunk 110B, theright visible light camera 114B can be formed on the circuit boards ofthe left chunk 110A, the temples 125A and 125B, or frame 105.

FIG. 2A is a right side view of another example hardware configurationof an eyewear device 100 utilized in the virtual wave creation system.As shown, the depth-capturing camera includes a left visible lightcamera 114A and a depth sensor 213 on a frame 105 to generate an initialdepth image of a sequence of initial depth images (e.g., in an initialvideo). Instead of utilizing at least two visible light cameras 114A and114B to generate the initial depth image, here a single visible lightcamera 114A and the depth sensor 213 are utilized to generate depthimages, such as the initial depth images. As in the example of FIGS.1A-D, wave effect selection from a user is applied to initial depthimages from the initial video to generate warped wave images of warpedwave video. The infrared camera 220 of the depth sensor 213 has anoutward facing field of view that substantially overlaps with the leftvisible light camera 114A for a line of sight of the eye of the user. Asshown, the infrared emitter 215 and the infrared camera 220 areco-located on the upper portion of the left rim 107A with the leftvisible light camera 114A.

In the example of FIG. 2A, the depth sensor 213 of the eyewear device100 includes an infrared emitter 215 and an infrared camera 220 whichcaptures an infrared image. Visible light cameras 114A and 114Btypically include a blue light filter to block infrared light detection,in an example, the infrared camera 220 is a visible light camera, suchas a low resolution video graphic array (VGA) camera (e.g., 640 × 480pixels for a total of 0.3 megapixels), with the blue filter removed. Theinfrared emitter 215 and the infrared camera 220 are co-located on theframe 105, for example, both are shown as connected to the upper portionof the left rim 107A. As described in further detail below, the frame105 or one or more of the left and right chunks 110A and 110B include acircuit board that includes the infrared emitter 215 and the infraredcamera 220. The infrared emitter 215 and the infrared camera 220 can beconnected to the circuit board by soldering, for example.

Other arrangements of the infrared emitter 215 and infrared camera 220can be implemented, including arrangements in which the infrared emitter215 and infrared camera 220 are both on the right rim 107A, or indifferent locations on the frame 105, for example, the infrared emitter215 is on the left rim 107B and the infrared camera 220 is on the rightrim 107B. However, the at least one visible light camera 114A and thedepth sensor 213 typically have substantially overlapping fields of viewto generate three-dimensional depth images. In another example, theinfrared emitter 215 is on the frame 105 and the infrared camera 220 ison one of the chunks 110A and 110B, or vice versa. The infrared emitter215 can be connected essentially anywhere on the frame 105, left chunk110A, or right chunk 110B to emit a pattern of infrared in the light ofsight of the eye of the user. Similarly, the infrared camera 220 can beconnected essentially anywhere on the frame 105, left chunk 110A, orright chunk 110B to capture at least one reflection variation in theemitted pattern of infrared light of a three-dimensional scene in thelight of sight of the eye of the user.

The infrared emitter 215 and infrared camera 220 are arranged to faceoutwards to pick up an infrared image of a scene with objects or objectfeatures that the user wearing the eyewear device 100 observes. Forexample, the infrared emitter 215 and infrared camera 220 are positioneddirectly in front of the eye, in the upper part of the frame 105 or inthe chunks 110A and 110B at either ends of the frame 105 with a forwardfacing field of view to capture images of the scene which the user isgazing at, for measurement of depth of objects and object features.

In one example, the infrared emitter 215 of the depth sensor 213 emitsinfrared light illumination in the forward facing field of view of thescene, which can be near-infrared light or other short-wavelength beamof low-energy radiation. Alternatively, or additionally, the depthsensor 213 may include an emitter that emits other wavelengths of lightbesides infrared and the depth sensor 213 further includes a camerasensitive to that wavelength that receives and captures images with thatwavelength. As noted above, the eyewear device 100 is coupled to aprocessor and a memory, for example in the eyewear device 100 itself oranother part of the virtual wave creation system. Eyewear device 100 orthe virtual wave creation system can subsequently process the capturedinfrared image during generation of three-dimensional depth images ofthe depth videos, such as the initial depth images from the initialvideo.

FIGS. 2B and 2C are rear views of example hardware configurations of theeyewear device 100, including two different types of image displays.Eyewear device 100 is in a form configured for wearing by a user, whichare eyeglasses in the example. The eyewear device 100 can take otherforms and may incorporate other types of frameworks, for example, aheadgear, a headset, or a helmet.

In the eyeglasses example, eyewear device 100 includes a frame 105including a left rim 107A connected to a right rim 107B via a bridge 106adapted for a nose of the user. The left and right rims 107A-B includerespective apertures 175A and 175B which hold a respective opticalelement 180A and 180B, such as a lens and a display device. As usedherein, the term lens is meant to cover transparent or translucentpieces of glass or plastic having curved and/or flat surfaces that causelight to converge/diverge or that cause little or no convergence ordivergence.

Although shown as having two optical elements 180A and 180B, the eyeweardevice 100 can include other arrangements, such as a single opticalelement or may not include any optical element 180A and 180B dependingon the application or intended user of the eyewear device 100. Asfurther shown, eyewear device 100 includes a left chunk 110A adjacentthe left lateral side 170A of the frame 105 and a right chunk 110Badjacent the right lateral side 170B of the frame 105. The chunks 110Aand 110B may be integrated into the frame 105 on the respective sides170A and 170B (as illustrated) or implemented as separate componentsattached to the frame 105 on the respective sides 170A and 170B.Alternatively, the chunks 110A and 110B may be integrated into temples(not shown) attached to the frame 105.

In one example, the image display of optical assembly 180A and 180Bincludes an integrated image display. As shown in FIG. 2B, the opticalassembly 180A and 180B includes a suitable display matrix 170 of anysuitable type, such as a liquid crystal display (LCD), an organiclight-emitting diode (OLED) display, or any other such display. Theoptical assembly 180A and 180B also includes an optical layer or layers176, which can include lenses, optical coatings, prisms, mirrors,waveguides, optical strips, and other optical components in anycombination. The optical layers 176A-N can include a prism having asuitable size and configuration and including a first surface forreceiving light from display matrix and a second surface for emittinglight to the eye of the user. The prism of the optical layers 176A-Nextends over all or at least a portion of the respective apertures 175Aand 175B formed in the left and right rims 107A-B to permit the user tosee the second surface of the prism when the eye of the user is viewingthrough the corresponding left and right rims 107A-B. The first surfaceof the prism of the optical layers 176A-N faces upwardly from the frame105 and the display matrix overlies the prism so that photons and lightemitted by the display matrix impinge the first surface. The prism issized and shaped so that the light is refracted within the prism and isdirected towards the eye of the user by the second surface of the prismof the optical layers 176A-N. In this regard, the second surface of theprism of the optical layers 176A-N can be convex to direct the lighttowards the center of the eye. The prism can optionally be sized andshaped to magnify the image projected by the display matrix 170, and thelight travels through the prism so that the image viewed from the secondsurface is larger in one or more dimensions than the image emitted fromthe display matrix 170.

In another example, the image display device of optical assembly 180Aand 180B includes a projection image display as shown in FIG. 2C. Theoptical assembly 180A and 180B includes a laser projector 150, which isa three-color laser projector using a scanning mirror or galvanometer.During operation, an optical source such as a laser projector 150 isdisposed in or on one of the temples 125A and 125B of the eyewear device100. Optical assembly 180A and 180B includes one or more optical strips155A-N spaced apart across the width of the lens of the optical assembly180A and 180B or across a depth of the lens between the front surfaceand the rear surface of the lens.

As the photons projected by the laser projector 150 travel across thelens of the optical assembly 180A and 180B, the photons encounter theoptical strips 155A-N. When a particular photon encounters a particularoptical strip, the photon is either redirected towards the user’s eye,or it passes to the next optical strip. A combination of modulation oflaser projector 150, and modulation of optical strips, may controlspecific photons or beams of light. In an example, a processor controlsoptical strips 155A-N by initiating mechanical, acoustic, orelectromagnetic signals. Although shown as having two optical assemblies180A and 180B, the eyewear device 100 can include other arrangements,such as a single or three optical assemblies, or the optical assembly180A and 180B may have a different arrangement depending on theapplication or intended user of the eyewear device 100.

As further shown in FIGS. 2B and 2C, eyewear device 100 includes a leftchunk 110A adjacent the left lateral side 170A of the frame 105 and aright chunk 110B adjacent the right lateral side 170B of the frame 105.The chunks 110A and 110B may be integrated into the frame 105 on therespective lateral sides 170A and 170B (as illustrated) or implementedas separate components attached to the frame 105 on the respective sides170A and 170B. Alternatively, the chunks 110A and 110B may be integratedinto temples 125A and 125B attached to the frame 105.

In one example, the image display includes a first (left) image displayand a second (right) image display. Eyewear device 100 includes firstand second apertures 175A and 175B which hold a respective first andsecond optical assembly 180A and 180B. The first optical assembly 180Aincludes the first image display (e.g., a display matrix 170A of FIG.2B; or optical strips 155A-N′ and a projector 150A of FIG. 2C). Thesecond optical assembly 180B includes the second image display e.g., adisplay matrix 170B of FIG. 2B; or optical strips 155AN″ and a projector150B of FIG. 2C).

FIG. 3 shows a rear perspective sectional view of the eyewear device ofFIG. 2A depicting an infrared camera 220, a frame front 330, a frameback 335, and a circuit board. It can be seen that the upper portion ofthe left rim 107A of the frame 105 of the eyewear device 100 includes aframe front 330 and a frame back 335. The frame front 330 includes afront-facing side configured to face outward away from the eye of theuser. The frame back 335 includes a rear-facing side configured to faceinward toward the eye of the user. An opening for the infrared camera220 is formed on the frame front 330.

As shown in the encircled cross-section 4-4 of the upper middle portionof the left rim 107A of the frame 105, a circuit board, which is aflexible printed circuit board (PCB) 340, is sandwiched between theframe front 330 and the frame back 335. Also shown in further detail isthe attachment of the left chunk 110A to the left temple 325A via a lefthinge 326A. In some examples, components of the depth sensor 213,including the infrared camera 220, the flexible PCB 340, or otherelectrical connectors or contacts may be located on the left temple 325Aor the left hinge 326A.

In an example, the left chunk 110A includes a chunk body 311, a chunkcap 312, an inward facing surface 391 and an outward facing surface 392(labeled, but not visible). Disposed inside the left chunk 110A arevarious interconnected circuit boards, such as PCBs or flexible PCBs,which include controller circuits for charging a battery, inwards facinglight emitting diodes (LEDs), and outwards (forward) facing LEDs.Although shown as being formed on the circuit boards of the left rim107A, the depth sensor 213, including the infrared emitter 215 and theinfrared camera 220, can be formed on the circuit boards of the rightrim 107B to captured infrared images utilized in the generation ofthree-dimensional depth images or depth videos, for example, incombination with right visible light camera 114B.

FIG. 4 is a cross-sectional view through the infrared camera 220 and theframe corresponding to the encircled cross-section 4-4 of the eyeweardevice of FIG. 3 . Various layers of the eyewear device 100 are visiblein the cross-section of FIG. 4 . As shown, the flexible PCB 340 isdisposed on the frame back 335 and connected to the frame front 330. Theinfrared camera 220 is disposed on the flexible PCB 340 and covered byan infrared camera cover lens 445. For example, the infrared camera 220is reflowed to the back of the flexible PCB 340. Reflowing attaches theinfrared camera 220 to electrical contact pad(s) formed on the back ofthe flexible PCB 340 by subjecting the flexible PCB 340 to controlledheat which melts a solder paste to connect the two components. In oneexample, reflowing is used to surface mount the infrared camera 220 onthe flexible PCB 340 and electrically connect the two components.However, it should be understood that through-holes can be used toconnect leads from the infrared camera 220 to the flexible PCB 340 viainterconnects, for example.

The frame front 330 includes an infrared camera opening 450 for theinfrared camera cover lens 445. The infrared camera opening 450 isformed on a front-facing side of the frame front 330 that is configuredto face outwards away from the eye of the user and towards a scene beingobserved by the user. In the example, the flexible PCB 340 can beconnected to the frame back 335 via a flexible PCB adhesive 460. Theinfrared camera cover lens 445 can be connected to the frame front 330via infrared camera cover lens adhesive 455. The connection can beindirect via intervening components.

FIG. 5 shows a rear perspective view of the eyewear device of FIG. 2A.The eyewear device 100 includes an infrared emitter 215, infrared camera220, a frame front 330, a frame back 335, and a circuit board 340. As inFIG. 3 , it can be seen in FIG. 5 that the upper portion of the left rimof the frame of the eyewear device 100 includes the frame front 330 andthe frame back 335. An opening for the infrared emitter 215 is formed onthe frame front 330.

As shown in the encircled cross-section 6-6 in the upper middle portionof the left rim of the frame, a circuit board, which is a flexible PCB340, is sandwiched between the frame front 330 and the frame back 335.Also shown in further detail is the attachment of the left chunk 110A tothe left temple 325A via the left hinge 326A. In some examples,components of the depth sensor 213, including the infrared emitter 215,the flexible PCB 340, or other electrical connectors or contacts may belocated on the left temple 325A or the left hinge 326A.

FIG. 6 is a cross-sectional view through the infrared emitter 215 andthe frame corresponding to the encircled cross-section 6-6 of theeyewear device of FIG. 5 . Multiple layers of the eyewear device 100 areillustrated in the cross-section of FIG. 6 , as shown the frame 105includes the frame front 330 and the frame back 335. The flexible PCB340 is disposed on the frame back 335 and connected to the frame front330. The infrared emitter 215 is disposed on the flexible PCB 340 andcovered by an infrared emitter cover lens 645. For example, the infraredemitter 215 is reflowed to the back of the flexible PCB 340. Reflowingattaches the infrared emitter 215 to contact pad(s) formed on the backof the flexible PCB 340 by subjecting the flexible PCB 340 to controlledheat which melts a solder paste to connect the two components. In oneexample, reflowing is used to surface mount the infrared emitter 215 onthe flexible PCB 340 and electrically connect the two components.However, it should be understood that through-holes can be used toconnect leads from the infrared emitter 215 to the flexible PCB 340 viainterconnects, for example.

The frame front 330 includes an infrared emitter opening 650 for theinfrared emitter cover lens 645. The infrared emitter opening 650 isformed on a front-facing side of the frame front 330 that is configuredto face outwards away from the eye of the user and towards a scene beingobserved by the user. In the example, the flexible PCB 340 can beconnected to the frame back 335 via the flexible PCB adhesive 460. Theinfrared emitter cover lens 645 can be connected to the frame front 330via infrared emitter cover lens adhesive 655. The coupling can also beindirect via intervening components.

FIG. 7 depicts an example of an emitted pattern of infrared light 781emitted by an infrared emitter 215 of the depth sensor 213. As shown,reflection variations of the emitted pattern of infrared light 782 arecaptured by the infrared camera 220 of the depth sensor 213 of theeyewear device 100 as an infrared image. The reflection variations ofthe emitted pattern of infrared light 782 is utilized to measure depthof pixels in a raw image (e.g., left raw image) to generatethree-dimensional depth images, such as the initial depth images of asequence of initial depth images (e.g., in an initial video).

Depth sensor 213 in the example includes the infrared emitter 215 toproject a pattern of infrared light and the infrared camera 220 tocapture infrared images of distortions of the projected infrared lightby objects or object features in a space, shown as scene 715 beingobserved by the wearer of the eyewear device 100. The infrared emitter215, for example, may blast infrared light 781 which falls on objects orobject features within the scene 715 like a sea of dots. In someexamples, the infrared light is emitted as a line pattern, a spiral, ora pattern of concentric rings or the like. Infrared light is typicallynot visible to the human eye. The infrared camera 220 is similar to astandard red, green, and blue (RGB) camera but receives and capturesimages of light in the infrared wavelength range. For depth sensing, theinfrared camera 220 is coupled to an image processor (element 912 ofFIG. 9 ) and the wave creation programming (element 945) that judge timeof flight based on the captured infrared image of the infrared light.For example, the distorted dot pattern 782 in the captured infraredimage can then be processed by an image processor to determine depthfrom the displacement of dots. Typically, nearby objects or objectfeatures have a pattern with dots spread further apart and far awayobjects have a denser dot pattern. It should be understood that theforegoing functionality can be embodied in programming instructions ofwave creation programming or application (element 945) found in one ormore components of the system.

FIG. 8A depicts an example of infrared light captured by the infraredcamera 220 of the depth sensor 213 with a left infrared camera field ofview 812. Infrared camera 220 captures reflection variations in theemitted pattern of infrared light 782 in the three-dimensional scene 715as an infrared image 859. As further shown, visible light is captured bythe left visible light camera 114A with a left visible light camerafield of view 111A as a left raw image 858A. Based on the infrared image859 and left raw image 858A, the three-dimensional initial depth imageof the three-dimensional scene 715 is generated.

FIG. 8B depicts an example of visible light captured by the left visiblelight camera 114A and visible light captured with a right visible lightcamera 114B. Visible light is captured by the left visible light camera114A with a left visible light camera field of view 111A as a left rawimage 858A. Visible light is captured by the right visible light camera114B with a right visible light camera field of view 111B as a right rawimage 858B. Based on the left raw image 858A and the right raw image858B, the three-dimensional initial depth image of the three-dimensionalscene 715 is generated.

FIG. 9 is a high-level functional block diagram of an example virtualwave creation system 900, which includes a wearable device (e.g., theeyewear device 100), a mobile device 990, and a server system 998connected via various networks. Eyewear device 100 includes an inputparameter processor 910 and a depth-capturing camera, such as at leastone of the visible light cameras 114A and 114B; and the depth sensor213, shown as infrared emitter 215 and infrared camera 220. Thedepth-capturing camera can alternatively include at least two visiblelight cameras 114A and 114B (one associated with the left lateral side170A and one associated with the right lateral side 170B).Depth-capturing camera generates initial depth images 961A-N of initialvideo 960, which are rendered three-dimensional (3D) models that aretexture mapped images of red, green, and blue (RGB) imaged scenes.

Mobile device 990 may be a smartphone, tablet, laptop computer, accesspoint, or any other such device capable of connecting with eyeweardevice 100 using both a low-power wireless connection 925 and ahigh-speed wireless connection 937. Mobile device 990 is connected toserver system 998 and network 995. The network 995 may include anycombination of wired and wireless connections.

Eyewear device 100 further includes two image displays of the opticalassembly 180A and 180B (one associated with the left lateral side 170Aand one associated with the right lateral side 170B). Eyewear device 100also includes image display driver 942, image processor 912, low-powercircuitry 920, and high-speed circuitry 930. Image display of opticalassembly 180A and 180B are for presenting images and videos, which caninclude a sequence of depth images, such as the initial depth images961A-N from the initial video 960. Image display driver 942 is coupledto the image display of optical assembly 180A and 180B to control theimage display of optical assembly 180A and 180B to present the videoincluding images, such as, for example, the initial depth images 961A-Nof initial video 960 and warped wave images 967A-N of warped wave video964. Eyewear device 100 further includes a user input device 991 (e.g.,touch sensor) to receive a wave effect selection from a user to applywaves to the presented initial video 960.

The components shown in FIG. 9 for the eyewear device 100 are located onone or more circuit boards, for example a PCB or flexible PCB, in therims or temples. Alternatively, or additionally, the depicted componentscan be located in the chunks, frames, hinges, or bridge of the eyeweardevice 100. Left and right visible light cameras 114A and 114B caninclude digital camera elements such as a complementarymetal-oxide-semiconductor (CMOS) image sensor, charge coupled device, alens, or any other respective visible or light capturing elements thatmay be used to capture data, including images of scenes with unknownobjects.

Eyewear device 100 includes a memory 934 which includes input parameterprogramming 911 and wave creation programming 945 to perform a subset orall of the functions described herein for wave creation, in which a waveeffect and input parameter selection from a user is applied to initialdepth images 961A-N to generate warped wave images 967A-N. As shown,memory 934 further includes a left raw image 858A captured by leftvisible light camera 114A, a right raw image 858B captured by rightvisible light camera 114B, and an infrared image 859 captured byinfrared camera 220 of the depth sensor 213.

As shown, eyewear device 100 includes an orientation sensor, whichincludes, for example, an inertial measurement unit (IMU) 972 asdepicted. Generally, an inertial measurement unit 972 is an electronicdevice that measures and reports a body’s specific force, angular rate,and sometimes the magnetic field surrounding the body, using acombination of accelerometers and gyroscopes, sometimes alsomagnetometers. In this example, the inertial measurement unit 972determines a head orientation of a wearer of the eyewear device 100which correlates to a camera orientation of the depth-capturing cameraof the eyewear device 100 when the associated depth image is captured,which is utilized in transforming a respective wave region of vertices966A-N, as explained below. The inertial measurement unit 972 works bydetecting linear acceleration using one or more accelerometers androtational rate using one or more gyroscopes. Typical configurations ofinertial measurement units contain one accelerometer, gyro, andmagnetometer per axis for each of the three axes: horizontal axis forleft-right movement (X), vertical axis (Y) for top-bottom movement, anddepth or distance axis for up-down movement (Z). The gyroscope detectsthe gravity vector. The magnetometer defines the rotation in themagnetic field (e.g., facing south, north, etc.) like a compass whichgenerates a heading reference. The three accelerometers detectacceleration along the horizontal (X), vertical (Y), and depth (Z) axesdefined above, which can be defined relative to the ground, the eyeweardevice 100, the depth-capturing camera, or the user wearing the eyeweardevice 100.

Memory 934 includes head orientation measurements which correspond toprincipal axes measurements on the horizontal axis (X axis), verticalaxis (Y axis), and depth or distance axis (Z axis) as tracked (e.g.,measured) by the inertial measurement unit 972. The head orientationmeasurements are utilized to determine alignment of the depth-capturingcamera, which can be used to identify a floor plane of the initial depthimages 961A-N. In certain applications of IMUs, the principal axes arereferred to as pitch, roll, and yaw axes.

Memory 934 further includes multiple initial depth images 961A-N, whichare generated, via the depth-capturing camera. Memory 934 furtherincludes an initial video 960 which includes a sequence of the initialdepth images 961A-N and associated time coordinates 963A-N. A flowchartoutlining functions which can be implemented in the wave creationprogramming 945 is shown in FIG. 11 . Memory 934 further includes a waveeffect selection 962 received by the user input device 991, which isuser input indicating that application of the wave effect on the initialvideo 960 is desired. The wave effect selection 962 impacts the strengthor degree to which the waves imparted on the initial video 960 warp theinitial depth images 961AN (e.g., by adjusting the amplitude orfrequency of the waves in response to an input parameter such as anaudio track). Additionally, the wave effect selection 962 may impact thecolor of vertices within the wave. Memory 934 further includestransformation matrices 965, wave regions of vertices 966A-N, affinitymatrices 968A-N, wave pattern 971, left and right rectified images969A-B (e.g., to remove vignetting towards the edge of the lens), and animage disparity 970, all of which are generated during image processingof the initial depth images 961A-N from the initial video 960 togenerate respective warped wave images 967A-N of the warped wave video964.

As shown in FIG. 9 , high-speed circuitry 930 includes high-speedprocessor 932, memory 934, and high-speed wireless circuitry 936. In theexample, the image display driver 942 is coupled to the high-speedcircuitry 930 and operated by the high-speed processor 932 in order todrive the left and right image displays of the optical assembly 180A and180B. High-speed processor 932 may be any processor capable of managinghigh-speed communications and operation of any general computing systemneeded for eyewear device 100. High-speed processor 932 includesprocessing resources needed for managing high-speed data transfers onhigh-speed wireless connection 937 to a wireless local area network(WLAN) using high-speed wireless circuitry 936. In some examples, thehigh-speed processor 932 executes an operating system such as a LINUXoperating system or other such operating system of the eyewear device100 and the operating system is stored in memory 934 for execution. Inaddition to any other responsibilities, the high-speed processor 932executing a software architecture for the eyewear device 100 managesdata transfers with high-speed wireless circuitry 936. In some examples,high-speed wireless circuitry 936 is configured to implement Instituteof Electrical and Electronic Engineers (IEEE) 802.11 communicationstandards, also referred to herein as Wi-Fi. In other examples, otherhigh-speed communications standards may be implemented by high-speedwireless circuitry 936.

Low-power wireless circuitry 924 and the high-speed wireless circuitry936 of the eyewear device 100 can include short range transceivers(Bluetooth™) and wireless wide, local, or wide area network transceivers(e.g., cellular or WiFi). Mobile device 990, including the transceiverscommunicating via the low-power wireless connection 925 and high-speedwireless connection 937, may be implemented using details of thearchitecture of the eyewear device 100, as can other elements of network995.

Memory 934 includes any storage device capable of storing various dataand applications, including, among other things, camera data generatedby the left and right visible light cameras 114A and 114B, infraredcamera 220, and the image processor 912, as well as images and videosgenerated for display by the image display driver 942 on the imagedisplays of the optical assembly 180A and 180B. While memory 934 isshown as integrated with high-speed circuitry 930, in other examples,memory 934 may be an independent standalone element of the eyeweardevice 100. In some such examples, electrical routing lines may providea connection through a chip that includes the high-speed processor 932from the image processor 912 or low-power processor 922 to the memory934. In other examples, the high-speed processor 932 may manageaddressing of memory 934 such that the low-power processor 922 will bootthe high-speed processor 932 any time that a read or write operationinvolving memory 934 is needed.

As shown in FIG. 9 , the processor 932 of the eyewear device 100 can becoupled to the depth-capturing camera (visible light cameras 114A and114B; or visible light camera 114A, infrared emitter 215, and infraredcamera 220), the image display driver 942, the user input device 991,and the memory 934. As shown in FIG. 10 , the processor 1030 of themobile device 990 can be coupled to the depth-capturing camera 1070, theimage display driver 1090, the user input device 1091, and the memory1040A. Eyewear device 100 can perform all or a subset of any of thefollowing functions described below as a result of the execution of thewave creation programming 945 in the memory 934 by the processor 932 ofthe eyewear device 100. Mobile device 990 can perform all or a subset ofany of the following functions described below as a result of theexecution of the wave creation programming 945 in the memory 1040A bythe processor 1030 of the mobile device 990. Functions can be divided inthe virtual wave creation system 900, such that the eyewear device 100generates the initial depth images 961A-N from the initial video 960,but the mobile device 990 performs the remainder of the image processingon the initial depth images 961A-N from the initial video 960 togenerate the warped wave images 967A-N of the warped wave video 964.

Execution of the wave creation programming 945 by the processor 932,1030 configures the virtual wave creation system 900 to performfunctions, including functions to generate, via the depth-capturingcamera, initial depth images 961A-N based on initial images 957A-N fromthe initial video 960. Each of the initial depth images 961A-N isassociated with a time coordinate on a time (T) axis for a presentationtime, based on, for example, initial images 957A-N, in the initial video960. Each of the initial depth images 961A-N is formed of a matrix ofvertices. Each vertex represents a pixel in a three-dimensional scene715. Each vertex has a position attribute. The position attribute ofeach vertex is based on a three-dimensional location coordinate systemand includes an X location coordinate on an X axis for horizontalposition, a Y location coordinate on a Y axis for vertical position, anda Z location coordinate on a Z axis for a depth position. Each vertexfurther includes one or more of a color attribute, a texture attribute,or a reflectance attribute.

Virtual wave creation system 900 presents, via the image display 180Aand 180B, 1080 the initial video 960. Eyewear device 100 receives, viathe user input device 991, 1091, a wave effect selection and an inputparameter selection from the user to apply waves to the presentedinitial video 960 responsive to the input parameter. Virtual wavecreation system 900 receives, via the user input device 991, 1091, thewave effect selection 962 and an input parameter selection from the userto apply waves to the presented initial video 960 responsive to theinput parameter.

In response to receiving the wave effect selection 962, based on, atleast, the associated time coordinate 963A-N of each of the initialdepth images 961A-N, virtual wave creation system 900 applies tovertices of each of the initial depth images 961A-N, a respectivetransformation function 965. The transformation function 965 transformsa respective wave region of vertices 966A-N grouped together along the Zaxis based on, at least, the associated time coordinate 963A-N of arespective initial depth image 961A-N and the input parameter. Thetransformation function 965 moves a respective Y location coordinate ofvertices in the respective wave region of vertices 966A-N verticallyupwards or downwards on the Y axis. Applying the transformation functioncreates a new modified set of vertices or a three-dimensional imagewithout a depth map. Additionally, the transformation function 965 maymodify the color components of the vertices within the wave responsivethe input parameter of another input parameter.

Virtual wave creation system 900 generates, for each of the initialdepth images 961A-N, a respective wave depth image 967A-N by applyingthe transformation function 965 to the respective initial depth image961A-N. The function of applying the respective transformation function965 to the respective initial depth image 961A-N can include multiplyingeach vertex in the respective wave region of vertices 966A-N of therespective initial depth image 961A-N by the transformation function 965to obtain a new Y location coordinate on the three-dimensional locationcoordinate system.

Virtual wave creation system 900 creates a warped wave video 964including the sequence of the generated warped wave images 967A-N.Virtual wave creation system 900 presents, via the image display 180Aand 180B, 1080, the warped wave video 964. The function of presentingvia the image display 180A and 180B, 1080, the warped wave video 964including the sequence of the generated warped wave images 967A-Npresents an appearance of rolling waves radiating radially from thedepth-capturing camera, radially from an object emitting a wave, oralong the Z axis of the warped wave images 967A-N of the warped wavevideo 964.

In one example of the virtual wave creation system 900, the processorcomprises a first processor 932 and a second processor 1030. The memorycomprises a first memory 934 and a second memory 1040A. The eyeweardevice 100 includes a first network communication 924 or 936 interfacefor communication over a network 925 or 937 (e.g., a wirelessshort-range network or a wireless local area network), the firstprocessor 932 coupled to the first network communication interface 924or 936, and the first memory 934 accessible to the first processor 932.Eyewear device 100 further includes wave creation programming 945 in thefirst memory 934. Execution of the wave creation programming 945 by thefirst processor 932 configures the eyewear device 100 to perform thefunction to generate, via the depth-capturing camera, the initial depthimages 961A-N from the initial video 960, the associated timecoordinates 963A-N, and the input parameter.

The virtual wave creation system 900 further comprises a host computer,such as the mobile device 990, coupled to the eyewear device 100 overthe network 925 or 937. The host computer includes a second networkcommunication interface 1010 or 1020 for communication over the network925 or 937, the second processor 1030 coupled to the second networkcommunication interface 1010 or 1020, and the second memory 1040Aaccessible to the second processor 1030. Host computer further includeswave creation programming 945 in the second memory 1040A.

Execution of the wave creation programming 945 by the second processor1030 configures the host computer to perform the functions to receive,via the second network communication interface 1010 or 1020, the initialvideo 960 over the network from the eyewear device 100. Execution of thewave creation programming 945 by the second processor 1030 configuresthe host computer to present, via the image display 1080, the initialvideo 960. Execution of the wave creation programming 945 by the secondprocessor 1030 configures the host computer to receive, via the userinput device 1091 (e.g., touch screen or a computer mouse), the waveeffect selection 962 from the user to apply waves to the presentedinitial video 960 responsive to an input parameter. Execution of thewave creation programming 945 by the second processor 1030 configuresthe host computer to, in response to receiving the wave effect selection962 based on, at least, the associated time coordinate 963A-N of each ofthe initial depth images 961A-N, 965 generate, for each of the initialdepth images 961A-N, the respective wave depth image 967A-N by applyingthe transformation function 965 to vertices of the respective initialdepth image 961A-N based on, at least, the Y and Z location coordinatesand the associated time coordinate 963A-N. Execution of the wavecreation programming 945 by the second processor 1030 configures thehost computer to create, the warped wave video 964 including thesequence of the generated warped wave images 967A-N. Execution of thewave creation programming 945 by the second processor 1030 configuresthe host computer to present, via the image display 1080, the warpedwave video 964.

In the example, the eyewear device 100 further includes an inertialmeasurement unit 972. Execution of the programming by the processorconfigures the virtual wave creation system 900 to perform the followingfunctions. Measure, via the inertial measurement unit 972, a rotation ofthe eyewear device 100 during capture of the initial depth images 961A-Nby the depth-capturing camera. For each of the initial depth images961A-N, determine a respective rotation matrix 973A-N to adjust X, Y,and Z location coordinates of the vertices based on the measuredrotation of the eyewear device 100 during capture. The respective warpedwave 967AN is generated by applying the rotation matrix 973A-N tovertices of the respective initial depth image 961A-N and then applyingthe transformation function 965.

In one example, applying the transformation function 965 to each initialdepth image moves the respective Y location coordinate of vertices inthe respective wave region of vertices 966A-N vertically upwards ordownwards on the Y axis to vertically fluctuate or oscillate therespective wave region of vertices 966A-N. The function of generating,for each of the initial depth images 961A-N, the respective wave depthimage 967A-N by applying the transformation function 965 to therespective initial depth image 961A-N vertically fluctuates oroscillates the respective wave region of vertices 966A-N and stores therespective initial depth image 961A-N with the vertical fluctuations oroscillations as the respective wave depth image 967A-N.

In some examples, the transformation function 965 moves the respective Ylocation coordinate of vertices in the respective wave region ofvertices 966A-N vertically upwards or downwards based on a wave pattern971. The wave pattern 971 provides an appearance of rolling wavesradiating radially from the depth-capturing camera, radially from anobject emitting a wave, or along the Z axis of the warped wave images967A-N of the warped wave video 964. This can provide a visual effectthat, for example, Elsa of Frozen ® is moving around the scenes of thewarped wave images 967A-N of the warped wave video 964 by manifestingitself with the ground having waves in response to an input parametersuch as an audio track.

Each initial depth image 961A-N includes a starting depth position onthe Z axis corresponding to a minimum depth of the respective initialdepth image 961A-N and an ending depth position on the Z axis having amaximum depth of the respective initial depth image 961AN based on aninput parameter (e.g., amplitude of an audio track). The function oftransforming the respective wave region of vertices 966A-N along the Zaxis based on, at least, the associated time coordinate 963A-N of therespective initial depth image 961A-N further includes the followingfunctions. For each of the sequence of the initial depth images 961A-N,iteratively transforming the respective wave region of vertices 966A-Nalong the Z axis based on progression of the associated time coordinate963A-N from the starting depth position to the ending depth position. Inresponse to reaching the ending depth position of the Z axis orexceeding a restart time interval of the wave pattern 971, restartingthe iterative selection of the respective wave region 966A-N at thestarting depth position.

In an example, an earlier initial depth image 961A is associated with anearlier time coordinate 963A on the time (T) axis for an earlierpresentation time in the initial video 960. An intermediate initialdepth image 961B is associated with an intermediate time coordinate 963Bon the time (T) axis for an intermediate presentation time after theearlier presentation time in the initial video 960. The function oftransforming the respective wave region of vertices 966A-N along the Zaxis based on, at least, the associated time coordinate 963A-N of therespective initial depth image 961A-N includes the following functions.Transforming a near range wave region of vertices 966A with nearer depthpositions grouped contiguously together along the Z axis for the earlierinitial depth image 961A based on the earlier time coordinate 963A.Transforming an intermediate range wave region of vertices 966B withintermediate depth positions grouped contiguously together along the Zaxis for the intermediate initial depth image 961B based on theintermediate time coordinate. The near range wave region of vertices966A is closer in depth along the Z axis than the intermediate rangewave region of vertices 966B.

In the example, a later initial depth image 961C is associated with alater time coordinate 963C on the time (T) axis for a later presentationtime after the intermediate presentation time of the intermediateinitial depth image 961B in the initial video 960. The function oftransforming the respective wave region of vertices 966A-N along the Zaxis based on, at least, the associated time coordinate 963A-N of therespective initial depth image 961A-N further includes transforming afar range wave region of vertices 966C with farther depth positionsgrouped contiguously together along the Z axis for the later initialdepth image 961C based on the later time coordinate 963C. The far rangewave region of vertices 966C is farther in depth along the Z axis thanthe intermediate range wave region of vertices 966C.

If the transformation matrices 965 are applied to a single vertex, aspike or pinch will occur. In order to generate a smooth (curvy) warpedwave images 967A-B, the affinity matrices 968A-N are computed as aregion of influence. For example, a polygon with a specific width andlength or a circle can be set with a specific radius. Then the amount oraffinity of each vertex to the polygon or the center of the circle (likea segmentation) is computed (e.g., utilizing edge detection), so eachvertex has a weight between zero and one as to how the vertex isinfluenced by the transformation function 965. Essentially each vertexmoves according to this weight. If the weight is one, the vertex istransformed according to the transformation function 965. If the weightis zero, the vertex does not move. If the weight is one-half, the vertexwill come halfway between the original position and the transformedposition.

Hence, execution of the wave creation programming 945 by the processor932, 1030 configures the virtual wave creation system 900 to performfunctions, including functions to compute a respective affinity matrix968A-N for the vertices of the respective initial depth image 961A-Nthat determines an influence weight of the transformation function 965on each of the vertices in the respective wave region of vertices966A-N. The influence weight is based on, at least, the verticalposition of the vertex. The function of generating, for each of theinitial depth images 961A-N, the respective wave depth image 967A-N byapplying the transformation function 965 to the respective initial depthimage 961A-N is further based on the computed respective affinity matrix968A-N. The influence weight is greater as a height of the vertexrelative to a floor plane of the respective initial depth image 961A-Ndecreases such that the transformation function 965 moves the Y locationcoordinate of the vertex vertically upwards on the Y axis to a greaterextent. The influence weight is lower as the height of the vertexrelative to the floor plane increases such that the transformationfunction 965 moves the Y location coordinate of the vertex verticallyupwards on the Y axis to a lesser extent.

In an example, virtual wave creation system 900 further includes aninertial measurement unit 972 like that shown for the eyewear device 100in FIG. 9 and the mobile device 990 in FIG. 10 . The function oftransforming the respective wave region of vertices 966A-N along the Zaxis based on, at least, the associated time coordinate 963A-N of therespective initial depth image 961A-N includes the following functions.Tracking, via the inertial measurement unit 972, a head orientation of ahead of a wearer of the eyewear device 100. The wearer of the eyeweardevice 100 is the user that is actually creating the warped wave video964 on a mobile device 990 or a different user that wore the eyeweardevice 100 when the initial video 960 was generated. Based on the headorientation, determining a floor plane of vertices which are contiguousalong the Z axis of the respective initial depth image 961A-N.Transforming the respective wave region of vertices 966A-N based on, atleast, the floor plane.

In the example, the function of tracking, via the inertial measurementunit 972, the head orientation of the head of the wearer includes thefollowing functions. First, measuring, via the inertial measurement unit972, the head orientation on the X axis, the Y axis, the Z axis, or thecombination thereof. Second, in response to measuring the headorientation, determining a deviation angle of the depth-capturing cameraon the X axis, the Y axis, the Z axis, or the combination thereof.Third, adjusting the floor plane of vertices based on the deviationangle, such as by re-orienting the vertices based on the deviation anglesuch that one axis, either the X axis, the Y axis, or the Z axis isperpendicular to the ground.

In one example, the depth-capturing camera of the eyewear device 100includes the at least two visible light cameras comprised of a leftvisible light camera 114A with a left field of view 111A and a rightvisible light camera 114B with a right field of view 111B. The leftfield of view 111A and the right field of view 111B have an overlappingfield of view 813 (see FIG. 8B). The depth-capturing camera 1070 of themobile device 990 can be similarly structured.

Generating, via the depth-capturing camera, the initial video 960including the sequence of initial depth images 961A-N and associatedtime coordinates 963A-N can include all or a subset of the followingfunctions. First, capturing, via the left visible light camera 114A, aleft raw image 858A that includes a left matrix of pixels. Second,capturing, via the right visible light camera 114B, a right raw image858B that includes a right matrix of pixels. Third, creating a leftrectified image 969A from the left raw image 858A and a right rectifiedimage 969B from the right raw image 858B that align the left and rightraw images 858A-B and remove distortion from a respective lens (e.g., atthe edges of the lens from vignetting) of each of the left and rightvisible light cameras 114A and 114B. Fourth, extracting an imagedisparity 970 by correlating pixels in the left rectified image 969Awith the right rectified image 969B to calculate a disparity for each ofthe correlated pixels. Fifth, calculating the Z location coordinate ofvertices of the initial depth image 961A based on at least the extractedimage disparity 970 for each of the correlated pixels. Sixth, orderingeach of the generated initial depth images 961A-N in the sequence fromthe initial video 960 based on a timestamp that is captured when theleft raw image 858A and the right raw image 858B are captured andsetting an associated respective time coordinate 963A-N of therespective initial depth image 961A-N to the timestamp.

In an example, the depth-capturing camera of the eyewear device 100includes the at least one visible light camera 114A and the depth sensor213 (e.g., infrared emitter 215 and infrared camera 220). The at leastone visible light camera 114A and the depth sensor 213 have asubstantially overlapping field of view 812 (see FIG. 8A). The depthsensor 213 includes an infrared emitter 215 and an infrared camera 220.The infrared emitter 215 is connected to the frame 105 or the temple125A and 125B to emit a pattern of infrared light. The infrared camera220 is connected to the frame 105 or the temple 125A and 125B to capturereflection variations in the emitted pattern of infrared light. Thedepth-capturing camera 1070 of the mobile device 990 can be similarlystructured.

Generating, via the depth-capturing camera, the initial depth image 961Acan include all or a subset of the following functions. First,capturing, via the at least one visible light camera 114A, a raw image858A. Second, emitting, via the infrared emitter 215, a pattern ofinfrared light 781 on a plurality of objects or object features locatedin a scene 715 that are reached by the emitted infrared light 781.Third, capturing, via the infrared camera 220, an infrared image 859 ofreflection variations of the emitted pattern of infrared light 782 onthe plurality of objects or object features. Fourth, computing arespective depth from the depth-capturing camera to the plurality ofobjects or object features, based on the infrared image 859 ofreflection variations. Fifth, correlating objects or object features inthe infrared image 859 of reflection variations with the raw image 858A.Sixth, calculating the Z location coordinate of vertices of the initialdepth image 961A based on, at least, the computed respective depth.

In one example, the user input device 991, 1091 includes a touch sensorincluding an input surface and a sensor array that is coupled to theinput surface to receive at least one finger contact inputted from auser. User input device 991, 1091 further includes a sensing circuitintegrated into or connected to the touch sensor and connected to theprocessor 932, 1030. The sensing circuit is configured to measurevoltage to track the at least one finger contact on the input surface.The function of receiving, via the user input device 991, 1091, the waveeffect selection 962 and input parameter identification from the userincludes receiving, on the input surface of the touch sensor, the atleast one finger contact inputted from the user.

A touch based user input device 991 can be integrated into the eyeweardevice 100. As noted above, eyewear device 100 includes a chunk 110A and110B integrated into or connected to the frame 105 on the lateral side170A and 170B of the eyewear device 100. The frame 105, the temple 125Aand 125B, or the chunk 110A and 110B includes a circuit board thatincludes the touch sensor. The circuit board includes a flexible printedcircuit board. The touch sensor is disposed on the flexible printedcircuit board. The sensor array is a capacitive array or a resistivearray. The capacitive array or the resistive array includes a grid thatforms a two-dimensional rectangular coordinate system to track X and Yaxes location coordinates.

Server system 998 may be one or more computing devices as part of aservice or network computing system, for example, that include aprocessor, a memory, and network communication interface to communicateover the network 995 with the mobile device 990 and eyewear device 100.Eyewear device 100 is connected with a host computer. For example, theeyewear device 100 is paired with the mobile device 990 via thehigh-speed wireless connection 937 or connected to the server system 998via the network 995.

Output components of the eyewear device 100 include visual components,such as the left and right image displays of optical assembly 180A and180B as described in FIGS. 2B and 2C (e.g., a display such as a liquidcrystal display (LCD), a plasma display panel (PDP), a light emittingdiode (LED) display, a projector, or a waveguide). Left and right imagedisplays of optical assembly 180A and 180B can present the initial video960 including the sequence of initial depth images 961A-N and the warpedwave images 967A-N of the warped wave video 964. The image displays ofthe optical assembly 180A and 180B are driven by the image displaydriver 942. Image display driver 942 is coupled to the image display tocontrol the image display to present the initial video 960 and thewarped wave video 964. The output components of the eyewear device 100further include acoustic components (e.g., speakers), haptic components(e.g., a vibratory motor), other signal generators, and so forth. Theinput components of the eyewear device 100, the mobile device 990, andserver system 998, may include alphanumeric input components (e.g., akeyboard, a touch screen configured to receive alphanumeric input, aphoto-optical keyboard, or other alphanumeric input components),point-based input components (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or other pointing instruments), tactile inputcomponents (e.g., a physical button, a touch screen that provideslocation and force of touches or touch gestures, or other tactile inputcomponents), audio input components (e.g., a microphone), biometriccomponents (e.g., a heart rate monitor) and the like.

Eyewear device 100 may optionally include additional peripheral deviceelements. Such peripheral device elements may include biometric sensors,additional sensors, or display elements integrated with eyewear device100. For example, peripheral device elements may include any I/Ocomponents including output components, motion components, positioncomponents, or any other such elements described herein.

For example, the biometric components include components to detectexpressions (e.g., hand expressions, facial expressions, vocalexpressions, body gestures, or eye tracking), measure biosignals (e.g.,blood pressure, heart rate, body temperature, perspiration, or brainwaves), identify a person (e.g., voice identification, retinalidentification, facial identification, fingerprint identification, orelectroencephalogram based identification), and the like. The motioncomponents include acceleration sensor components (e.g., accelerometer),gravitation sensor components, rotation sensor components (e.g.,gyroscope), and so forth. The position components include locationsensor components to generate location coordinates (e.g., a GlobalPositioning System (GPS) receiver component), WiFi or Bluetooth™transceivers to generate positioning system coordinates, altitude sensorcomponents (e.g., altimeters or barometers that detect air pressure fromwhich altitude may be derived), orientation sensor components (e.g.,magnetometers), and the like. Such positioning system coordinates canalso be received over wireless connections 925 and 937 from the mobiledevice 990 via the low-power wireless circuitry 924 or high-speedwireless circuitry 936.

FIG. 10 is a high-level functional block diagram of an example of amobile device 990 that communicates via the virtual wave creation system900 of FIG. 9 . Mobile device 990 includes a user input device 1091 andan input parameter processor 1092 to receive a wave effect selection 962from a user to apply waves to the initial depth images 961A-N of thepresented initial video 960 to generate warped wave images 967A-N of thewarped wave video 964.

Mobile device 990 includes a flash memory 1040A which includes inputparameter programming 911 and wave creation programming 945 to performall or a subset of the functions described herein for wave creation, inwhich a wave effect and input parameter selection from a user is appliedto an initial video 960 to create a warped wave video 964. As shown,memory 1040A further includes a left raw image 858A captured by leftvisible light camera 114A, a right raw image 858B captured by rightvisible light camera 114B, and an infrared image 859 captured byinfrared camera 220 of the depth sensor 213. Mobile device 990 caninclude a depth-capturing camera 1070 that comprises at least twovisible light cameras (first and second visible light cameras withoverlapping fields of view) or at least on visible light camera and adepth sensor with substantially overlapping fields of view like theeyewear device 100. When the mobile device 990 includes components likethe eyewear device 100, such as the depth-capturing camera, the left rawimage 858A, the right raw image 858B, and the infrared image 859 can becaptured via the depth-capturing camera 1070 of the mobile device 990.

Memory 1040A further includes multiple initial depth images 961A-N,which are generated, via the depth-capturing camera of the eyeweardevice 100 or via the depth-capturing camera 1070 of the mobile device990 itself. Memory 1040A further includes an initial video 960 whichincludes a sequence of the initial depth images 961A-N and associatedtime coordinates 963A-N. A flowchart outlining functions which can beimplemented in the wave creation programming 945 is shown in FIG. 11 .Memory 1040A further includes a wave effect selection 962 received bythe user input device 1091, which is user input indicating thatapplication of the wave effect on the initial video 960 is desired. Insome examples, the wave effect selection 962 may impact the strength ordegree to which the waves imparted on the initial video 960 warp theinitial depth images 961A-N (e.g., by adjusting the amplitude orfrequency of the waves) and/or colors responsive to the input parameter.Memory 1040A further includes transformation matrices 965, wave regionsof vertices 966A-N, affinity matrices 968A-N, wave pattern 971, left andright rectified images 969A-B (e.g., to remove vignetting towards theedge of the lens), and an image disparity 970, all of which aregenerated during image processing of the initial depth images 961A-Nfrom the initial video 960 to generate respective warped wave images967A-N of the warped wave video 964.

As shown, the mobile device 990 includes an image display 1080, an imagedisplay driver 1090 to control the image display, and a user inputdevice 1091 similar to the eyewear device 100. In the example of FIG. 10, the image display 1080 and user input device 1091 are integratedtogether into a touch screen display.

Examples of touch screen type mobile devices that may be used include(but are not limited to) a smart phone, a personal digital assistant(PDA), a tablet computer, a laptop computer, or other portable device.However, the structure and operation of the touch screen type devices isprovided by way of example; and the subject technology as describedherein is not intended to be limited thereto. For purposes of thisdiscussion, FIG. 10 therefore provides block diagram illustrations ofthe example mobile device 990 having a touch screen display fordisplaying content and receiving user input as (or as part of) the userinterface.

The activities that are the focus of discussions here typically involvedata communications related to processing initial depth images 961A-Nfrom the initial video 960 to generate warped wave images 967A-Nresponsive to an input parameter in order to create the warped wavevideo 964 in the portable eyewear device 100 or the mobile device 990.As shown in FIG. 10 , the mobile device 990 includes at least onedigital transceiver (XCVR) 1010, shown as WWAN XCVRs, for digitalwireless communications via a wide area wireless mobile communicationnetwork. The mobile device 990 also includes additional digital oranalog transceivers, such as short range XCVRs 1020 for short-rangenetwork communication, such as via NFC, VLC, DECT, ZigBee, Bluetooth™,or WiFi. For example, short range XCVRs 1020 may take the form of anyavailable two-way wireless local area network (WLAN) transceiver of atype that is compatible with one or more standard protocols ofcommunication implemented in wireless local area networks, such as oneof the Wi-Fi standards under IEEE 802.11 and WiMAX.

To generate location coordinates for positioning of the mobile device990, the mobile device 990 can include a global positioning system (GPS)receiver. Alternatively, or additionally the mobile device 990 canutilize either or both the short range XCVRs 1020 and WWAN XCVRs 1010for generating location coordinates for positioning. For example,cellular network, WiFi, or Bluetooth™ based positioning systems cangenerate very accurate location coordinates, particularly when used incombination. Such location coordinates can be transmitted to the eyeweardevice over one or more network connections via XCVRs 1010, 1020.

The transceivers 1010, 1020 (network communication interface) conformsto one or more of the various digital wireless communication standardsutilized by modern mobile networks. Examples of WWAN transceivers 1010include (but are not limited to) transceivers configured to operate inaccordance with Code Division Multiple Access (CDMA) and 3rd GenerationPartnership Project (3GPP) network technologies including, for exampleand without limitation, 3GPP type 2 (or 3GPP2) and LTE, at timesreferred to as “4G.” For example, the transceivers 1010, 1020 providetwo-way wireless communication of information including digitized audiosignals, still image and video signals, web page information for displayas well as web related inputs, and various types of mobile messagecommunications to/from the mobile device 990 for wave creation.

Several of these types of communications through the transceivers 1010,1020 and a network, as discussed previously, relate to protocols andprocedures in support of communications with the eyewear device 100 orthe server system 998 for wave creation, such as transmitting left rawimage 858A, right raw image 858B, infrared image 859, initial video 960,initial depth images 961A-N, time coordinates 963A-N, warped wave video964, and warped wave images 967A-N. Such communications, for example,may transport packet data via the short range XCVRs 1020 over thewireless connections 925 and 937 to and from the eyewear device 100 asshown in FIG. 9 . Such communications, for example, may also transportdata utilizing IP packet data transport via the WWAN XCVRs 1010 over thenetwork (e.g., Internet) 995 shown in FIG. 9 . Both WWAN XCVRs 1010 andshort range XCVRs 1020 connect through radio frequency (RF)send-and-receive amplifiers (not shown) to an associated antenna (notshown).

The mobile device 990 further includes a microprocessor, shown as CPU1030, sometimes referred to herein as the host controller. A processoris a circuit having elements structured and arranged to perform one ormore processing functions, typically various data processing functions.Although discrete logic components could be used, the examples utilizecomponents forming a programmable CPU. A microprocessor for exampleincludes one or more integrated circuit (IC) chips incorporating theelectronic elements to perform the functions of the CPU. The processor1030, for example, may be based on any known or available microprocessorarchitecture, such as a Reduced Instruction Set Computing (RISC) usingan ARM architecture, as commonly used today in mobile devices and otherportable electronic devices. Other processor circuitry may be used toform the CPU 1030 or processor hardware in smartphone, laptop computer,and tablet.

The microprocessor 1030 serves as a programmable host controller for themobile device 990 by configuring the mobile device 990 to performvarious operations, for example, in accordance with instructions orprogramming executable by processor 1030. For example, such operationsmay include various general operations of the mobile device, as well asoperations related to the wave creation programming 945 andcommunications with the eyewear device 100 and server system 998.Although a processor may be configured by use of hardwired logic,typical processors in mobile devices are general processing circuitsconfigured by execution of programming.

The mobile device 990 includes a memory or storage device system, forstoring data and programming. In the example, the memory system mayinclude a flash memory 1040A and a random access memory (RAM) 1040B. TheRAM 1040B serves as short term storage for instructions and data beinghandled by the processor 1030, e.g., as a working data processingmemory. The flash memory 1040A typically provides longer term storage.

Hence, in the example of mobile device 990, the flash memory 1040A isused to store programming or instructions for execution by the processor1030. Depending on the type of device, the mobile device 990 stores andruns a mobile operating system through which specific applications,including wave creation programming 945, are executed. Applications,such as the wave creation programming 945, may be a native application,a hybrid application, or a web application (e.g., a dynamic web pageexecuted by a web browser) that runs on mobile device 990 to create thewarped wave video 964 from the initial video 960 based on the receivedwave effect selection 962. Examples of mobile operating systems includeGoogle Android, Apple iOS (I-Phone or iPad devices), Windows Mobile,Amazon Fire OS, RIM BlackBerry operating system, or the like.

It will be understood that the mobile device 990 is just one type ofhost computer in the virtual wave creation system 900 and that otherarrangements may be utilized. For example, a server system 998, such asthat shown in FIG. 9 , may create waves in the initial video 960 aftergeneration of the initial depth images 961A-N, via the depth-capturingcamera of the eyewear device 100.

FIG. 11 is a flowchart of a method with steps that can be implemented inthe virtual wave creation system 900 to apply waves to the initial depthimages 961A-N from the initial video 960 to generate the warped waveimages 967A-N to create the warped wave video 964. Because the blocks ofFIG. 11 were already explained in detail previously, repetition of allof the details is avoided here.

Beginning in block 1100, the method includes generating, via thedepth-capturing camera, a sequence of initial depth images 961A-N frominitial images 957A-N of an initial video 960.

Proceeding now to block 1110, the method further includes determining,for each of the initial depth images 961A-N, a respective rotationmatrix 973A-N. The respective rotation matrix 973A-N is to adjust X, Y,and/or Z location coordinates of the vertices based on detected rotationof the depth-capturing camera. For example, the rotation matrix 973A-Ncan be a 2×2 or 3×3 matrix with X, Y, and/or Z axis position adjustmentsor angles to normalize the vertices in the captured initial depth images961A-N with the floor plane to correct for camera rotation.

Continuing to block 1120, the method further includes generating, foreach of the initial depth images 961A-N, a respective warped wave image967A-N by applying the respective rotation matrix 973A-N and thetransformation function 965 that is responsive to an input parameter tovertices of a respective initial depth image 961A-N. The transformationfunction 965 transforms a respective wave region of vertices 966A-Ngrouped together along a Z axis based on, at least, the associated timecoordinate 963A-N of the respective initial depth image 961A-N and aparameter of the audio track (e.g., amplitude samples). Thetransformation function 965 moves a respective Y location coordinate ofvertices in the respective wave region of vertices 966A-N verticallyupwards or downwards on the Y axis based on a wave pattern 971.Additionally, the transformation function 965 may alter the colors ofthe vertices responsive to the same or a different input parameter.Moving now to block 1130, the method further includes creating, a warpedwave video 964 including the sequence of the generated warped waveimages 967A-N.

Finishing now in block 1140, the method further includes presenting, viaan image display 180A and 180B or 1080, the warped wave video 964. Thestep of presenting via the image display 180A and 180B or 1080, thewarped wave video 964 including the sequence of the generated warpedwave images 967A-N presents an appearance of a rolling wave radiallyfrom the depth-capturing camera, radially from an object emitting awave, or along a Z axis of the warped wave images 967A-N of the warpedwave video 964.

FIGS. 12A and 12B illustrate an example of a first raw image 858Acaptured by one of the visible light cameras 114A and 114B andapplication of a transformation function 965 to a first wave region ofvertices 966A of a generated first initial depth image 961A,respectively. A first time coordinate 963A set to 0.00 seconds isassociated with the first raw image 858A during capture, and thereforethe corresponding first initial depth image 961A and the first wavedepth image 967A are also associated with the first time coordinate 963Aof 0.00 seconds. In FIG. 12A, the first raw image 858A is depicted ascaptured by one of the visible light cameras 114A and 114B before anyimage processing (e.g., rectification, etc.). Thus, the first raw image858A has a fisheye appearance, resulting from vignetting by the visiblelight camera 114A and 114B. The first raw image 858A includes varioustwo-dimensional pixels with X and Y location coordinates on an X axis1205 and a Y axis 1210. The corresponding first initial depth image of961A of the sequence of initial depth images 961A-N in the initial video960 is subsequently generated using techniques described previously.

In FIG. 12B, a Z axis 1215 is depicted as being overlaid on thegenerated first wave depth image 967A of the created warped wave video964. A floor plane 1220 of the first wave depth image 967A is contiguousalong the Z axis 1215. In addition to the orientation sensor techniquesdisclosed above (e.g., utilizing an inertial measurement unit 972) toidentify the floor plane 1220, a heuristic can be utilized that assumesthe floor plane 1220 is somewhere between 5 feet and 6 feet from thevertical position of the depth-capturing camera that generated the firstinitial depth image 961A. This assumes a human of average height worethe eyewear device 100 when the first raw image 858A was captured anddid not skew or rotate his/her head. In this instance, the person wasstanding five to six feet above floor level (e.g., ground level). InFIG. 12B, the application of the first transformation function 965A onthe first wave region 966A is depicted, and this results in waves whichappear to be in the near range as a result of the first wave region 966Abeing in close range (e.g., short depth/distance) on the Z axis 1215.

FIGS. 13A and 13B illustrate an example of a second raw image 858B of avideo captured by one of the visible light cameras 114A and 114B andapplication of the transformation function 965 responsive to an originselection and an input parameter. In FIGS. 13A and 13B, the originselection corresponds to the position of a character 1302, e.g., Elsafrom Frozen ®, which may be positioned by a user using an input device.In FIG. 13B, the application of the transformation function 965, whichis responsive to the input parameter, is depicted, and this results inwaves which appear to emanate from the selected origin in an outwardradial pattern, which is seen in the deformation of the tubes 1304. Theamount of deformation is based on a parameter of the input (e.g.,amplitude or an audio track) at a playing time corresponding to theplaying time of the video.

FIG. 14 illustrates waves emanating from an origin 1402 corresponding toa character 1302. Moving the character (e.g., using a finger on adisplay) results in the origin changing to match the new location of thecharacter. FIG. 15 illustrates an augmented reality implementation inwhich waves through a virtual structure (e.g., pyramid 1502) arepresented to the wearer over an actual structure (e.g., pyramid 1504) inthe wearers field of view.

Any of the wave creation functionality described herein for the eyeweardevice 100, mobile device 990, and server system 998 can be embodied inone or more applications as described previously. According to someexamples, “function,” “functions,” “application,” “applications,”“instruction,” “instructions,” or “programming” are program(s) thatexecute functions defined in the programs. Various programming languagescan be employed to create one or more of the applications, structured ina variety of manners, such as object-oriented programming languages(e.g., Objective-C, Java, or C++) or procedural programming languages(e.g., C or assembly language). In a specific example, a third-partyapplication (e.g., an application developed using the ANDROID™ or IOS™software development kit (SDK) by an entity other than the vendor of theparticular platform) may be mobile software running on a mobileoperating system such as IOS™, ANDROID™, WINDOWS® Phone, or anothermobile operating systems. In this example, the third-party applicationcan invoke API calls provided by the operating system to facilitatefunctionality described herein.

Hence, a machine-readable medium may take many forms of tangible storagemedium. Non-volatile storage media include, for example, optical ormagnetic disks, such as any of the storage devices in any computer(s) orthe like, such as may be used to implement the client device, mediagateway, transcoder, etc. shown in the drawings. Volatile storage mediainclude dynamic memory, such as main memory of such a computer platform.Tangible transmission media include coaxial cables; copper wire andfiber optics, including the wires that comprise a bus within a computersystem. Carrier-wave transmission media may take the form of electric orelectromagnetic signals, or acoustic or light waves such as thosegenerated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer may read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to aprocessor for execution.

The scope of protection is limited solely by the claims that now follow.That scope is intended and should be interpreted to be as broad as isconsistent with the ordinary meaning of the language that is used in theclaims when interpreted in light of this specification and theprosecution history that follows and to encompass all structural andfunctional equivalents. Notwithstanding, none of the claims are intendedto embrace subject matter that fails to satisfy the requirement ofSections 101, 102, or 103 of the Patent Act, nor should they beinterpreted in such a way. Any unintended embracement of such subjectmatter is hereby disclaimed.

Except as stated immediately above, nothing that has been stated orillustrated is intended or should be interpreted to cause a dedicationof any component, step, feature, object, benefit, advantage, orequivalent to the public, regardless of whether it is or is not recitedin the claims.

It will be understood that the terms and expressions used herein havethe ordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”“includes,” “including,” or any other variation thereof, are intended tocover a non-exclusive inclusion, such that a process, method, article,or apparatus that comprises or includes a list of elements or steps doesnot include only those elements or steps but may include other elementsor steps not expressly listed or inherent to such process, method,article, or apparatus. An element preceded by “a” or “an” does not,without further constraints, preclude the existence of additionalidentical elements in the process, method, article, or apparatus thatcomprises the element.

Unless otherwise stated, any and all measurements, values, ratings,positions, magnitudes, sizes, and other specifications that are setforth in this specification, including in the claims that follow, areapproximate, not exact. Such amounts are intended to have a reasonablerange that is consistent with the functions to which they relate andwith what is customary in the art to which they pertain. For example,unless expressly stated otherwise, a parameter value or the like mayvary by as much as ± 10% from the stated amount.

In addition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in various examples for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimed examplesrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, the subject matter to be protected liesin less than all features of any single disclosed example. Thus, thefollowing claims are hereby incorporated into the Detailed Description,with each claim standing on its own as a separately claimed subjectmatter.

While the foregoing has described what are considered to be the bestmode and other examples, it is understood that various modifications maybe made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that they may be appliedin numerous applications, only some of which have been described herein.It is intended by the following claims to claim any and allmodifications and variations that fall within the true scope of thepresent concepts.

What is claimed is:
 1. A virtual wave creation system comprising: aneyewear device including a camera; a biometric component configured todevelop a biometric input parameter; an image display for presenting aninitial video including initial images; a processor coupled to thecamera and the biometric component, the processor configured to:present, via the image display, the initial video; receive at least oneinput parameter to apply waves to the presented initial video, the atleast one input parameter including the biometric input parameter;generate a sequence of initial depth images from respective initialimages in the initial video; generate, based on the input parameters,for each of the initial depth images, a respective warped wave image byapplying a transformation function that is responsive to the biometricinput parameter to the respective initial depth image; create, a warpedwave video including the sequence of the generated warped wave images;and present, via the image display, the warped wave video.
 2. The systemof claim 1, wherein the biometric component is a heart rate monitor andthe biometric input parameter is a heart rate obtained from the heartrate monitor.
 3. The system of claim 1, wherein: the input parametersare sampled and sample parameters in each of the samples are averaged toobtain the parameter for that respective sample.
 4. The system of claim3, wherein the parameter for each respective sample is applied to thetransformation function.
 5. The system of claim 4, wherein the parameteris amplitude.
 6. The system of claim 1, wherein presenting, via theimage display, the warped wave video including the sequence of thegenerated warped wave images presents an appearance of a wavefrontadvancing radially from an object.
 7. The system of claim 6, wherein theobject is a character and wherein moving the character changes an originof the appearance of the wavefront.
 8. The system of claim 1, whereinthe biometric component is an expression biometric component configuredto detect a biometric expression input parameter as the biometric inputparameter.
 9. The system of claim 8, wherein the biometric expressioninput parameter is at least one of a hand expression input parameter, afacial expression input parameter, a vocal expression input parameter, abody gesture expression input parameter, or an eye tracking expressioninput parameter.
 10. The system of claim 1, wherein the biometriccomponent is a biosignal measurement component configured to detect thebiometric input parameter.
 11. The system of claim 10, wherein thebiometric input parameter is at least one of blood pressure, heart rate,body temperature, perspiration, or brain waves.
 12. A virtual inputparameter-based wave creation method comprising steps of: presenting,via an image display, an initial video; receiving, via a biometriccomponent, at least one input parameter including a biometric inputparameter; generating a sequence of initial depth images from initialimages of the initial video; generating, based on the input parametersfrom a user, for each of the initial depth images, a respective warpedwave image by applying a transformation function that is responsive tothe biometric input parameter to the respective initial depth image;creating a warped wave video including the sequence of the generatedwarped wave images; and presenting, via the image display, the warpedwave video.
 13. The method of claim 12, further comprising: sampling theinput parameters; and averaging sample parameters in each of the samplesto obtain the parameter for that respective sample.
 14. The method ofclaim 13, further comprising: applying the parameter for each respectivesample to the transformation function.
 15. The method of claim 12,wherein presenting, via the image display, the warped wave videoincluding the sequence of the generated warped wave images comprises:presenting an appearance of a rolling wave advancing radially from anobject.
 16. The method of claim 15, wherein the object is a characterand wherein moving the character changes an origin of the appearance ofwaves in the warped wave video.
 17. The method of claim 12, wherein thebiometric component is a biosignal measurement component configured todetect the biometric input parameter; wherein the biometric inputparameter is at least one of blood pressure, heart rate, bodytemperature, perspiration, or brain waves.
 18. A non-transitorycomputer-readable medium comprising instructions which, when executed bya processor, cause an electronic system to: generate a sequence ofinitial depth images from initial images of an initial video; receive atleast one input parameter including a biometric input parameter receivedfrom a biometric component; generate, based on the input parameters, foreach of the initial depth images, a respective warped wave image byapplying a transformation function that is responsive to the biometricinput parameter to the respective initial depth image; create, a warpedwave video including the sequence of the generated warped wave images;and present, via an image display, the warped wave video.
 19. Thenon-transitory computer-readable medium of claim 18, wherein theinstructions further cause the electronic system to: sample the inputparameters; and average sample parameters in each of the samples toobtain the parameter for that respective sample.
 20. The non-transitorycomputer-readable medium of claim 18, wherein presenting, via the imagedisplay, the warped wave video including the sequence of the generatedwarped wave images presents an appearance of a wavefront advancingradially from an object, wherein the object is a character and whereinmoving the character changes an origin of the appearance of thewavefront.